Real-time Collaboration Tools for Digital Ink

The importance and use of digital ink in formulas and diagrams is well recognized, and currently utilized by scientific and technical users of pen-enabled computers. Similarly, there is a growing recognition as to the utility of collaborative tools that allow multiple users to edit a document, either simultaneously or asynchronously. Finally, more people are using real-time communication tools to enhance problem-solving teamwork and productivity in commercial and educational environments. But the ability to perform all of these tasks simultaneously is poorly supported by current software solutions. In this paper, we explain how real-time collaboration tools for digital ink should provide the potential for increased technical productivity, along with the results of functional software testing that was performed by several students and a teacher in efforts to identify those features most necessary for their effective use. INTRODUCTION Typically, in mathematics classes, problem-solving sessions replace the usual notion of a science laboratory for the purposes of hands-on learning. In those sessions, an instructor (often a teaching assistant) helps students with solutions and facilitates a discussion. However, more recently, it has been strongly recommended that the joint activity of solving problems in small groups is a superior method of learning, often called active learning for obvious reasons. The skills and knowledge obtained in this type of social interaction often seem to stick in a more permanent fashion [4]. In our investigation, we specifically focused on problems in science and mathematics that would typically be solved by drawing a diagram and/or writing a formula. The usual method of interaction between students in this case would be to write using pencil and/or pen on a piece of paper laid horizontally on a table, or chalk or dry-erase marker vertically on a blackboard or whiteboard respectively. While these forms of interaction are well understood as providing a high degree of bandwidth and interactivity, they require the participants to be present in the same room (disallowing remote collaboration), and do not provide a convenient way of saving work from one session to the next (unless an electronic scanning board is utilized). Although electronic whiteboards do exist to provide remote collaboration, their size, expense, and lack of portability make them difficult to use in the typical classroom. During the summer of 2006, I was involved in supervising a project that I initiated to study collaborative problem solving, and investigative the utility (if any) of utilizing tablet computers to assist in this activity. Its purpose was to investigate the potentials and limitations of using tablet computers for collaborative problem solving and classroom presentations, and the project was supported by the Cascade Mentoring Program at my institution, funded by the Howard Hughes Medical Institute. In order to study the potential advantages that electronic communication and storage might provide, we employed a high school teacher and two high school students, working with a combination of portable and desktop tablet computers. As mentioned, the types of problems chosen were specifically designed to take advantage of the ability to draw diagrams and write formulas directly on the screen, and the hardware we used specifically supported this natural form of interaction. * appears in The Journal of Computing Sciences in Colleges, Volume 25, Number 3, January 2010. BACKGROUND The project started with the premise that all types of problem solving are facilitated by two fundamental principles: the ability to communicate ideas with other interested parties in real-time; and the ability to store the results of those interactions for long-term reference. In this paper, we will refer to these fundamental principles as collaboration and persistence, and study the ways in which these principles are enhanced (or hindered) by the hardware and software technologies that were tested. But before we go into the details of our preliminary results, it would be useful to describe from a more abstract point of view the ideas of collaboration and persistence. We will assume that the primary aspect of problem solving (at least in science and mathematics) involves the building of mental models. Transmitting those models through time and/or space is a fundamental challenge to any individual or group working on a technical solution. Those models often take the form of a combination of textually annotated diagrams and formulas, together with the ancillary oral explanations that people often add to such forms of written communication. This information is most readily transmitted by pen and voice, though it can also be captured on a tablet computer equipped with a microphone. REAL-TIME COLLABORATION Communication can be defined as the transmission of information from one point in space (here) to another (there). Typically, participants engage in the same room, where voice transmission takes place face to face. The sharing of written information can occur through a (mobile) piece of paper, a (fixed) board, or a combination of these two instruments. However, neither permits remote collaboration between participants that cannot directly see or hear each other. Communication allows information transfer between people separated in space. LONG-TERM PERSISTENCE Storage can be defined as the transmission of information from one point in time (now) to a later point (then). When someone records written information on a piece of paper or board, those marks are retained until the piece of paper is lost or destroyed, or the board is erased. On the other hand, oral information is not retained in a conversation unless a specific technology for voice recording is employed, such as a microphone attached to an electronic recorder or computer. But even in the ideal situation where the paper is not misplaced and the board is not erased, it is difficult for each participant to retain a written copy of those interactions. Storage makes possible access to information separated by time. PAPER One of the supreme advantages of paper is that it combines both the options of long-term persistence and real-time collaboration. People can annotate an existing document, carry that information from place to place, and even collaborate with someone else using the same piece of paper, which retains a record of that information interaction. However, editing information (such as moving things around) or erasing is tedious at best (with pencil) and impossible at worst (with pen). Moreover, this “technology” does not permit remote collaboration, nor does it allow each of the participants to each retain a written copy of their interactions (without a copy machine). Using a tablet computer could solve both of these problems, allowing multiple participants to collaborate from separate locations, and in addition allow for the long-term storage and organization of notes related to specific problem solutions obtained as the result of that collaboration. SIMULTANEITY IN SPACE AND TIME The final ingredient that needs to be specified is known as deixis -the proof or demonstration of an idea by simultaneously speaking about it and pointing to it -a ubiquitous technique in nearly all forms of instruction. What is really going on here is that in order to transmit information about a mental model from one individual to another, the teacher (the one who is doing the transmitting) must indicate which aspect of a diagram or formula he or she is currently talking about. By pointing to the object under consideration, the teacher is cementing a correspondence between the temporal information being transmitted (by voice) and the spatial information being transmitted (by pen). This is why teaching at a board has been such a successful format, and why most professors (particularly in math and science) will refuse to teach in a classroom without ample board space. Instructors typically use their hand, a piece of chalk, or a long stick as a pointer to draw attention to the specific area of the board that is under consideration at a given moment. As an aside, the best scientific articles utilize a similar technique to connect the written text of a discussion (which conveys the temporal information) with specific numbers or letters called out on an illustration (conveying the spatial information). Scientific American magazine has become famous for its colorful illustrations which, when describing a process, often include specifically numbered areas with individual captions. The importance of deixis in online mathematics education is well recognized [5]. In what follows, we will outline the approach we took to analyzing various hardware and software configurations, and how they assisted or fell short of the goals articulated above. We found that of all the implementations, it was most difficult to perform both deixis (for the teacher) and organization (for the student), and that these items were the most challenging to achieve from a performance point of view. PRIOR WORK One of the first equipment designs to incorporate nearly all the important ideas required for remote collaboration using digital ink was developed by Hewlett-Packard. Originally known as the DeskSlate [6], this device permitted two parties (and no more) to collaborate with voice and digital ink over an ordinary telephone line. In real-time, each person could hear the other’s voice (like an ordinary telephone conversation), and simultaneously see each other’s scribbling on a jointly shared screen. Both parties had their own individual pens, which when they weren’t inking served as pointers. When a session was over, each participant could save the resulting screen images, organize them, and even print them for archival purposes. The machine was marketed as the Omnishare [7], but it did not catch on because of its very high price, limited market appeal, and poor penetration (remember that you could only use its capabilities with the person you were calling if they also had one of these hard-to-find and expensive devices). However, in terms of persistence and collaboration, together with the simultaneous voice and support for dual deixis pointers, this is the design to beat. Hewlett-Packard’s Omnishare [8] (reproduced by permission of photographer) HARDWARE Digital inking requires an absolute positioning device, so special hardware is required because a mouse is a relative positioning device (i.e. it senses motion). The least expensive way to do this is to use a graphics tablet which senses the position of an RF pen above its surface, and translates that into a cursor position on the screen. Although this is a perfectly functional method, these can take quite a bit of getting used to since they demand a fair degree of hand-eye coordination. A simpler, but more expensive solution is to use an integrated tablet monitor, in which the digital pen “writes” directly on the screen. These are most commonly available as special notebook computers known as Tablet PCs, and also as specialized desktop monitors used in graphics applications. We chose to use a pair of tablet computers made by Sony around 2002, model LX-920. These unique desktops [9] included an integrated pen-tablet monitor, and ran the ordinary version of Windows XP (not the special Tablet PC version included with notebooks). Sony’s PCV-LX920 desktop tablet computer Because the monitor allows the screen angle to be adjusted from vertical to nearly horizontal, this hardware provided an almost ideal platform to evaluate digital inking. We also used one of the first generation Tablet PCs made by Hewlett Packard, the TC1000. The HP TC1000 Tablet PC This hybrid (as it’s known in the industry) is convenient because the tablet can be detached from its keyboard for carry-around portability, or perched on its keyboard for use on a table. Its small size and wireless capability was also important because it can be used by the instructor to see what students are doing on their computers while moving about the classroom. Apple currently provides little support for digital ink on their computers, despite the fact that their portable Newton MessagePad was one of the first devices to support digital ink in a robust way across its platform. In its last incarnation before being discontinued, the handwriting recognition was reportedly excellent. But despite this, Apple has never since provided substantial support for incorporating pen-based ink. SOFTWARE A fair number of pieces of software are ink-enabled, such as the well-known Microsoft PowerPoint, making it useful in a one-way mode that would typically be used in a lecture-style classroom for teacher-tostudent interaction. Also, because only one computer is being used, the cursor serves well as a pointer for deixis. A lesser-known piece of ink-enabled software called Microsoft Office Document Imaging allows one to electronically annotate any document that can be printed or scanned, but it too does not provide a mechanism for collaboration. However, both pieces of software allow annotations to be saved, and therefore do permit persistence. In the context of teaching, this permits the instructor pick up where he or she left off last. This is a significant difference between ink-enabled overlays (which work with virtually any piece of software) and ink-enabled software in which the capability to store annotations digitally is built into the underlying file format. Since our work requires remote collaboration between two computers, we focused on ink-enabled software that could work in a two-way mode to allow for peer-to-peer interaction required in problemsolving sessions between students. Identifying such software and determining which features were crucial formed the focus of our work. We examined five programs: OneNote (an optional part of Microsoft Office); DyKnow; Classroom Presenter; Conference XP; and NetMeeting (a little known part of Microsoft Windows). The DyKnow software has been the focus of many articles highlighting its usefulness in classroom presentation and group work, e.g. [1, 2]. The two high school students worked together solving many sets of mathematics and physics problems, including logic puzzles and brain teasers (selected by the college student mentor). Their experiences formed the basis of our findings. Experiences using Tablet PCs in a variety of other scenarios are the subject of an entire book [3]. FINDINGS Multiple panels (i.e. virtual whiteboards) were extremely helpful, especially if there was an option to synchronize them (ensuring that students were literally on the same page). Sitting across from each other, students had to rely on a remote pointer for deixis. Since this was the most poorly implemented feature in general, they preferred to sit side by side (though it was pointed out that there may be a gender basis for this, since both students were women). With the exception of OneNote, which had a vast feature set and capability for organization, none of the other pieces of software provided a way for students to retain their handwritten work in a coherent fashion (i.e. organized persistence). Support for remote pointers was nonexistent in Conference XP, though it did allow for voice communication. In Classroom Presenter, panels were not editable during a presentation, but it did allow inking from multiple people at once. NetMeeting featured both application sharing and voice communication, but only between two users (i.e. no conference calls). However, there was whiteboard support for multiple users, including simultaneous inking and text editing, with movable hands that served as repositionable remote pointers (though they often were buggy). As mentioned, OneNote featured powerful organizational tools, and a unique file sharing mode where multiple users could simultaneously text edit and ink to the same open file, as long as they didn’t modify the same information simultaneously. But highly inked pages had long latency and refresh delays. Remote pointing was implemented with disappearing ink – information could be circled or underlined, and seconds later that annotation would be gone. This was acceptable, but certainly not ideal for proper communication in complicated problems. DyKnow was the most capable piece of software – not surprising since it was specifically designed for collaborative instruction. Its strengths were in its ability to manage the instructorto-student interaction, but the remote pointer displayed a very high degree of latency. CONCLUSIONS NetMeeting is a free piece of software included on every Windows computer, and can perform some of the most basic tasks related to remote collaboration, including application sharing, voice communication, and basic digital inking with pointers. Although multiple users can take advantage of the whiteboard features, there’s no facility for instructor management, nor is there ability to organize session notes. This software was most useful for taking advantage of real-time interactivity. OneNote had superior organizational features, but its difficulty with lots of ink on a page, and the use of disappearing ink instead of a remote pointer, limited its utility for high bandwidth real-time interactivity required in solving technical problems. This software seemed better suited for asynchronous collaboration in which organized persistence was far more important than real-time communication. DyKnow provided a strong suite of tools for instructor-student and student-student interaction, making it the most promising piece of classroom-based software. Remote pointer lag and the limited ability to organize session notes made this product more difficult to use for remote collaborative work requiring high interactivity or information management. But, it was the only piece of software evaluated that was designed by a company specifically dedicated to educational technology. The most challenging (and in many ways the most important) feature was to facilitate deixis, with individual pointers representing remote cursors (something the OmniShare had no trouble with even though it was developed and based on a technologically challenged platform compared with today's devices). People naturally need to point to items within equations or figures in order to convey their "point". Until this feature can be implemented with a high degree of reliability and speed (enough to keep up with a person's voice), remote collaboration involving digital inking will remain noticeably inferior to live contact between parties at the same board or screen. ACKNOWLEDGEMENTS The author appreciates financial support for this project from HHMI, along with the participation of high school teacher Alan Bronstein together with his students Angel Feng and Katherine Sioson from Central High School in Philadelphia, along with the mentoring from Haverford College student Michael Jablin. Additional support from DyKnow, allowing us to use the software gratis, is also appreciated.